Color sensor data is a measurement of the intensity of light that is absorbed by a pixel of a sensor. Since different colors absorb lights are different rates, the intensity at which light is absorbed by the pixels of the sensor is an indication of the color of light that has been absorbed. Relative absorption levels can thus be represented as digital voltages in digital format to represent objects. Data from single color sensor devices capable of displaying color images, that employ one or more of video frame as well as still image capture must be interpolated from raw sensor image data that has been derived from color filter array (CFA), a physical material placed over sensors to make sure that only the correct color information is provided to the sensors, to a color space to generate a full resolution color image. Such image devices can include, but should not be limited to, digital cameras, video phones, and camcorders.
There are many color spaces that may be used, including the Red Green Blue color space, denoted by RGB, a color space defined by Magenta, Green and Yellow colors or MGY, YUV, YCbCr, wherein the Y luma component denotes the brightness of an image and the two color difference components Cb and Cr denote the chroma information of that image. Color space data may optionally be down-sampled, also referred to as sampling, subsampling or decimating. Common down-sampling indicators include 4:4:4, 4:2:2, 4:1:1, 4:2:0, etc. 4:4:4 RGB denotes a 2×2 array of RGB pixels that occupy a total of 12 bytes, assuming 8 bits per sample. YCbCr components can be formed from the RGB color space; without subsampling from 4:4:4 RGB, this is referred to as 4:4:4 YCbCr. 4:2:2 is a type of sampling used in YCbCr digital video in which chroma (CbCr) components are subsampled by a factor of 2 along the horizontal axis and in which chroma samples are cosited with alternate luma samples. In an 8-bit system using 4:2:2 YCbCr coding, for example, the 2×2 array would occupy 8 bytes and the aggregate data capacity would be 16 bits per pixel. This type of subsampling is often used for studio digital video. In 4:1:1 sampling, used in some digital video systems, chroma components are subsampled by a factor of four horizontally. The uncompressed data capacity is 12 bits per pixel. In 4:2:0 YCbCr sampling, often used in JPEG and MPEG systems, Cb and Cr are each subsampled by a factor of 2 along the horizontal and vertical axes. Chroma samples are interstitially sited, vertically between lines and are horizontally sited halfway between alternate luma samples (unlike the cositing of chroma samples in 4:2:2 sampling). The uncompressed aggregate data capacity of 4:2:0 sampling is 12 bits per pixel.
As can be seen from the traditional image acquisition and processing chain illustrated in FIGS. 2A-3B, there are many steps required to generate usable output data, in a desired color space, from raw sensor data. Many of the functions illustrated in the FIGS. 1A-2B are time-consuming and processor intensive. Referring now to the functional flow 100 of FIG. 1A, raw sensor data 110 is interpolated to one or more color planes in a first color space at 112. For instance, the raw sensor data may be CFA data that is interpolated to the three planes of color in the RGB color space. At 114, color correction of the interpolated color plane data is performed by assistance of a matrix multiply 116. Color correction serves to make the color more saturated so as to be perceived in a more pleasant manner. Color correction at 114, is performed in color space 1. Next, at 118 gamma correction is performed to compensate for the gamma function of the display device upon which the image is to be displayed, such as a monitor's liquid crystal display (LCD). Conversion from the first color space to a second color space takes place, aided by a conversion matrix. The conversion matrix 122 is usually a standard conversion matrix or multiplier that is used to perform the conversion, such as a standard conversion matrix capable of converting RGB color space data to YCbCr color space data. Often the conversion matrix will use conversion values that require the use hardware capable of performing multiplier functionality, in addition to just add and shift functionality, as will be demonstrated in FIGS. 2A and 3A below. At 124, the color space 2 data may optionally be down-sampled to the desired format. The output color space two data 126 is thus generated after the conversion at 120 or the optional subsampling at 124.
FIG. 1B illustrates a color space flow 150 that tracks the effects of functional flow 100 on the color spaces. CFA raw sensor data 150 is interpolated into the first color space, which, in this illustration, comprises three color planes 160, 162, 164. These planes of color space 1 are next converted to planes 170, 172, 174 of the second color space following the color correction and gamma correction detailed at 114 and 118 above in the first color space. Finally, the color space 2 data can be optionally down-sampled, i.e. sampled or decimated, to obtain corresponding planes 180, 182, 184 of down-sampled, converted data in color space 2.
FIGS. 2A-3B illustrate the traditional image acquisition and procession chain for two specific color space examples. Again, interpolation, color correction and gamma correction of the color data is performed in color space 1, while any optional down-sampling is performed in color space 2 after conversion from color space 1 to color space 2 has taken place. In these figures, RGB CFA raw sensor data is provided to the processing chain, but the down-sampling of converted YCbCr data is processed differently.
Referring now to FIGS. 2A-2B, an exemplary example of interpolation, correction and conversion of color data from the RGB color space to the YCbCr color space, followed by decimation to 4:2:2 YCbCr, is demonstrated. As noted above, 4:2:2 sampling, in which chroma components are subsampled by a factor of 2 along the horizontal axis coincident with alternate luma samples, is often used for digital video applications. Functional flow 200 starts when raw sensor data 210 is provided for interpolation to three planes, one each of Red, Green, Blue, in the RGB color space 1 at 212. Color correction 214 of the RGB plane data is performed using matrix multiply 216, followed by gamma correction 218. Next, the RGB color data is converted 220 using the conversion matrix illustrated as 222. The use of the parameter values shown in conversion matrix 222 requires the use of additional conversion hardware, in the form of multipliers, not just adders and shifters.
The color space flow 250 of FIG. 2B reflects changes to the color space caused by the functional flow 200 of FIG. 2A. Raw CFA RGB data 255 is interpolated into the RGB data color planes, one each for Red, Green, and Blue, and shown as 260, 262, 264. Following color correction and gamma correction in the first RGB color space, the data is converted to 4:4:4 YCbCr color data planes 270, 272, 274. This converted color data may optionally be subsampled to 4:2:2 YCbCr color data planes 280, 282, 284, as shown.
Referring now to FIGS. 3A-3B, an exemplary example of interpolation, correction and conversion of color data from the RGB color space to the YCbCr color space, followed by decimation to 4:2:0 YCbCr, will now be discussed. 4:2:0 is often used in JPEG and MPEG applications and the Cb and Cr components are each subsampled by a factor of 2 for both the horizontal and vertical axes chroma or Y samples are sited interstitially in a vertical fashion halfway between the lines and are sited horizontally halfway between alternate luma samples. Functional flow 300 starts when raw CFA sensor data 310 is provided for interpolation to three planes, one each of Red, Green, Blue, in the RGB color space 1 at 312. Color correction 314 of the RGB plane data is performed using matrix multiply 316, followed by gamma correction 318. Next, the RGB color data is converted 320 using the conversion matrix illustrated as 322. The use of the parameter values shown in conversion matrix 322 requires the use of additional conversion hardware, in the form of multipliers, not just adders and shifters.
The color space flow 350 of FIG. 3B reflects changes to the color space caused by the functional flow 300 of FIG. 3A. Raw CFA RGB data 355 is interpolated into the RGB data color planes, one each for Red, Green, and Blue, and shown as RGB color planes 360, 362, 364. Following color correction and gamma correction in the first RGB color space, the data is converted to 4:4:4 YCbCr color data planes 370, 372, 374. This converted color data may optionally be subsampled to 4:2:0 YCbCr color data planes 380, 382, 384, as shown.
As can be seen in FIGS. 1A-3B, a primary concern is the interpolation of color sensor data to the RGB color space. For those cases where YCbCr data is required, the interpolated RGB data is converted to YCbCr data via a matrix multiply and then decimated to generate the desired data, such as the 4:2:2 or 4:2:0 sampling demonstrated in FIGS. 2 and 3. This process is inefficient because the three full planes of color data in the RGB color space must be interpolated and converted, and ultimately some of the calculated data is discarded.
The manner is which interpolation may be performed in the traditional image acquisition and processing approach of FIGS. 1A-3B will now be explored. Interpolation of color data from a single sensor traditionally operates by moving a sliding window over the sensor data and computing a single pixel value for that window. This approach has centered the window used in the interpolation process over a single pixel and then generated a single pixel output value. Most color filter arrays have a 2×2 color filter pattern, such as a Bayer pattern for RGB color data, for example, and consequently four different interpolation algorithms are required depending on which of the four pixels the window centered. Other color data patterns may of course be used, such as a striped pattern. Other color filter patterns, such as Cyan, Yellow, Magenta, could also be used in place of the RGB color filter pattern shown in this example. A 3×3 window is used to compute a single interpolated value for the center pixel of that window. Interpolation by this pixel-by-pixel approach is thus not only very time-consuming; it also represents a major commitment in processing resources.
FIG. 4 illustrates a simplified traditional pixel-centered interpolation using a 3×3 pixel window centered over a 2×2 Bayer color filter pattern, shown as a Green pixel in the upper left position, a Red pixel in the upper right position, a Blue pixel in the lower left position, and a Green pixel in the lower right position. A corresponding YCbCr pattern could be used if working in the YCbCr color space. RGB data from the pixels surrounding the centered pixel are used to interpolate RGB output data for the center pixel. In the figure, the 3×3 pixel window is centered over the center pixel in the window, G5. The single output pixel value for pixel G5, referred to as P5, is given by:P5=f(G1, R2, G3, B4, G5, B6, G7, R8, G9)
This indicates that the Red, Green and Blue components of pixels G5, given as R5, G5, and B5, respectively, might be given as:R5=(R2+R8)/2G5=(G1+G3+G7+G9+G5)/5B5=(B4+B6)/2
It is noted that in addition to the interpolation described above, other interpolation methods may be used, including, but not limited to, nearest neighbor replication, bilinear interpolation, smooth hue transition interpolation, binary interpolation, for example.
This approach will only provide output color data for the pixel centered in the window, G5, so the pixel window has to be moved to a new pixel in order to generate interpolated output color data for that pixel. This is illustrated in FIGS. 5A-5D, in which the pixel window is centered over four different pixels of interest G7, B8, R12, and G13 to interpolate output color data for them. The 2×2 Bayer color filter pattern has been maintained for simplicity and is repeated over 25 pixels as shown. The four interpolated pixels for which color data output values are generated, G7, B8, R12, and G13, require positioning the window four times, with the pixel window being centered over four different patterns. In FIG. 5A, the pixel pattern, starting in the upper left corner of the 3×3 window and reading across rows from left to right, is G R G B G B G R G. In FIG. 5B, the pattern is R G R G B G R G R. In FIG. 5C, the pattern is B G B G R G B G B and in FIG. 5D, the pattern is G B G R G R G B G. Consequently, four different interpolation algorithms are required, depending upon which of the four pixels the pixel window is centered.
The requirement of four different algorithms for interpolation is time-consuming and can represent a major drain on the computation powers required to carry them out. There is also a corresponding increase in the amount of hardware that is required to carry out the instructions of these algorithms. Moreover, the many functions provided in the traditional image acquisition and processing chain, of which interpolation is but one, are each performed in serial fashion. These many steps are time consuming and use up processor resources. It would be advantageous in the art to have a less complex method for image acquisition and processing.